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METHOD AND DEVICE FOR FORMING AN IMAGE 

The invention relates to a method for fcrmmg en image and to an image sensor with 
an active area containing a plurality of pixels. It makes possible a wide dynamic 
range but obviates the need for a storage of complete images. It reduces tempond 
aliasing to ft minimum and eliminates spatial alia«ng. 

5 Image sensing devices can be realized with semiconductors based on their capability 
to convert locally impinging light energy into a proportional amount of electronic 
charge. More specifically, if a picture element is exposed during time 7 to the local 
light power Pu a charge signal Q is created according to the equation 

Q'^-PlYT (I) 

10 where y denotes the conversion efficiency of photons into electronic charge, which 

strongly depends on the wavdcngthspectmm of the inconMnglightr^^ 

often called pbotocharge, can be stored in a two-dimensional array of charge storage 
devices such as a reverse biased diode (as in photodiode-based image sensors) or 
such as a pre-charged metal-oxide-scmiconductor capacitance (as in cherge-coupled 
15 devices, CCDs). 

Due to the limited storage capacity of these storage devices, the ratio of maximum 
storable photocharge to photocharge detection noise, called dynamic range, is also 
limited. In typical CCD or photodiode image sensors, the available dynamic range is 
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of the order of TOOO; 1. Unfortunately, natural scenes (whm there is no control of 
lighting conditions) and indoor scenes with highly varying illununation have a 
dynamic range of between 100*000:1 and 1^*000:1. 



The dynamic range of an image sensor can be increased by making use of the 
S exposure time dependence shown in Equation (1). The patents US-S» 144,442, US* 
5,168,532, US-5,309»243 and US-5,S17,242 describe methods based on the^ 
acquisition of two or more images, each with its individual CTcposure time. At least 
one complete image has to be stored with these methods, preferentially in digital 
form by making use of a flrame-store. This results in a complex and cost-intensive 
10 system. Moreover, the two or more images with different exposure times cannot be 
taken concurrently, therefbre not representuig the same moving scenes at di£forent 
exposure levels but rather at different points in time. Consequently, such methods 
exhibit undesirable temporal aliasing. 

This problem can be overcome by a method described in US-S,483,365 and US- 
IS 5,789,737. The approach taken in US-5,483,365 consists of using alternate image 

sensor rows for different exposure times. US-5,789J37 teaches the use of several 

picture elements (pixels), each with its own sensitivity. In both cases, the brightness 
information may be acquired concurrently in time but not at the identical geom^ca! 

pixel location. This implies spatial undersampjing and aliasing, which is particularly 
20 undesirable in the case of so-called highlights, i.e., localized very bright pixel values 

usually caused by specular reflections at objects in the scene. 



Once a plurality of images have been taken at different exposure times, they have to 
be fUsed or merged to form one single piece of pixel information of wide dynamic 
range. Patents US^,647.975, US-S, 168,532 and US.5,671,013 teach that the 
25 information is copied from the most suitable of the images, according to some 
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selection rule* TUs value is then multiplied with a suitable fiictor that corrects for the 
respective exposure time. This method worics welt only for ideal image sensors with 
completely linear pixel behavior, irrespective of iUumination level and exposure 
time. In practical im^e sensors, this is not true, and the resulting response curve 
S (output vahie vs. iliumination levels) ahows discondnuities. This is particularly 
distutbing if the resulting images are processed fiirther, leading to &lse contours and 
erroneous contrast values. An improvement is taught by US-5, SI 7,242 claiming an 
algorithm where the output value at each pixel site is calculated in a certain 
brightness range as a linear combination of the values at two different exposure 
10 times, corrected by an appropriate &ctor that compensates for the different exposure 
times. In all these methods, either complete images have to be stored^ or complex and 
surflice-intensive electronic circuitry in each pixel is required* 

It is the aim of the invention to overcome the aforementioned disadvimtages of the 
prior-art methods and image sensora. In particular, the Invention shall make possible 
IS a wide dynamic range but obviate the need for a storage of complete Images. It shall, 
moreover, reduce temporal and spatial aliasing to a minimum The problem is solved 
by the invention as defined in the mdependent claims. 

The invention is based on the local acquisition of pixel info rma tion for different 
exposure times, following Equation (1), The problems of the requiremem fbr 

20 complete image storage and of tempoml or spatial sampling at difTereot points in 
time or in space are overcome by using a type of image sensors in which a subset of 
pixels (e.g., a single pixel or a row of pixels) can be individually reset and read out. 
A prefbrr^ embodiment is an active pixel sensor (APS) pixel that can be fabricated 
in standard CMOS technology. Such an APS is described for example in E.R. 

25 Possum, "Active Pixel Sensors: Are CCD*s Dinosaurs?'', ?roc. SPIE 1900 (1993) 2- 
14. 
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The method according to the invention for forming an image um an image semor 
with an active axiea containing a plurality of pixels. The method comprides 

a) in n interrogation runs performed on a first subset of pixels, where » is an integer 
and /t ;^ 2, resetting the first subset of pixels, ejqposing the first subset of pixels 

S and reading out the output valueCs) of the first subset of pixels; 

b) combining said output values into a first combined output value; 

c) repeating steps a) and b) for at least one second subset of pixels. 

In order to obtain a complete image information, steps a) and b) are preferably 
repeated until each pixel has been read out at least once* 

10 The subset is understood to contain less pixels than the whole image sensor. The 
si^set may be, e.g., a row, a column or a single pixel of the image sensor All subsets 
pref«-ably have the same number of pixels. The subsets are preferably defined before 
performing step a) of the method by partitioning the active area of the image sensor. 
The processing of one subset of pixels may temporally overlap with the processing of 

1 S the following subset of pixels. 

In a preferred embodlnwot of the invention, n « 2 for ail subsets of pixels. i.e.« each 
svbs&L is exposed twice, with a ioqger andTihorter ensure time. Al&kt reset 
conunand is Umei to one pixel or all pixels in one row. After a firsts long exposure 
time Txmt^ output value(8) Piont of the pixel or all pixels in this row is/are read out 

20 and immediately stored in analog or digital form. Immediately afterwards^ the pixel 
or row of pixels is reset and exposed during a second, short exposure time 7ih«t- The 
output value(8) i\hort of the pixei or row of pixels is/are read out. This information is 
either again stored in analog or digital fashion* or it can immediately be processed 
together with the previously stored pixel or row-of-pixei wtput values, according to 

2S the procedure given below. Depending on the desired vahie of 71m^ the pixel or row 
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of pixels can immediately be reset for a maximuin value of exposure time* or it/they 
can be reset at a later time. After this pixel or row of pixels has been reset and read 
twice» the next pixel or row of pixels is interrogated in the same &8hion. 

in the method according to the Invention, the timing used to reset and interrogate the 
S sequence of pixels in an Image sensor is important The ratio i - 71e^r«iurt is chosen 
to increase significantly the dynamic range of the image sensor, insuring, however, 
an overlap between the images. For example, the dynamic range of an image sensor 
can be increased by the method according to the invention ftom 70 dB to 110 dB 
using a ratio of r - 100. During the long int^ration time, many pixels or rows of 

10 pixels can be interrogated and reset a first time, exposed during a second (much 
shorter) time interval^ interrogated and reset a second time. Pre^bly, these 
operations are peribnned on all other pixels or rows of pixels during the long 
integration time of one pixel or row of pixels, so that the ratio / » TumJTAvt is equal 
to the total number of pixels or rows of pixels of the image sensor. Compared with 

IS the state of the ait, this has three advantages. Firstly, at most two rows of pixel data 
have to be stored at one time, and it is completely unnecessary to acquits and store 
complete images before processing and merging the data. Secondly, the short 
exposure time occurs immediately after the long one, effectively assuring a quasi- 
instantaneous exposure. Thirdly, the same pixel location is employed to acqiaire 

20 — image data fbr-thc diffident c5qK>8ure times and no spati^^ 



During the ftrst, long exposure, dark objects are detected, whereas very bright objects 
are not, due to saturation of the corresponding pixds; during the second, short 
exposure, bright objects are detected, whereaa very dark objects are not, due to a too 
weak signal According to the invention, the output signals of both exposures are 
2S suitably combined in order to increase the dynamic range of the image sensor. Due to 
imperfections in the practical realirations of the above-described method and device, 
the two pixel output values P^^ and Piwg measured for the two exposure times will 
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not be rtlated ideally through the nttio / - W7tot> It., be close to but not 

identical with t^P^. For this reason we need an algoiithm that combined the two 
values in a matiner that does not lead to artifacts in subsequent image processing and 
interpretation procedures. For this, it is important to find a method that ensures a 
5 monotonic smo^h transition from values taken with long to values talten with short 
exposure tunes. 

The invention encompasses the following method to arrive at this goal. For each 
pixel or row of pixels, the two pixel output values P\ang and ?.h«i are processed into a 
single value Foui, using a merging function J?^i»x:;/: 

The merging function ^hX^ is preftrably truly monotonic, continuous and 
continuously differentiable in both independent variables xt and xi. It should have the 
following properties: 

(i) fficux^ = xi for aci 5 

15 (u) /(xx,x^^xx {oTXx^x^vihere0^xum<x^ 
(iiO /(kux^ increases truly monotonically 

in xi and JCi &mjow^ xi-< j^ 

In other words, the merging function/i^^i,*;^ preftrably obeys the following rules: 

(i) Preference is given to the output value xi obtained from the longer exposure 
20 when the output values xu xi or an approbate combination of the output 

values xi. x% e.g., thdr average (xi + X2)/2, lie beneath a given lower limit 

(ii) Pr^ence is given to the output value xa obtained from the shorter exposure 
when the output values Xu xa or an i^}propriatc combination of the output 

25 N'alues xi, xa, e.g., their average (xi + xa)/2, lie above a given upper limit x^ 
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(in) The merging function f^ux^) iacreaaes truly monotonically in the out{nit 
values xu X2 when the output valiies Xh xi lie between the lower Hmh xiow and 
the upper limit x^p. 

The parani0ta^x,9» xw are chosen accordix^ tothe following criteria. 
S « The lower limit xtow is given by the minimum illumination required for the short 

exposure time. For lower tUuminaticn, the sensor output corresponds eftsentially 

to Doise and cannot be interpreted. 
• The upper limit Xup corresponds to the output value computed for maximum 

allowable light iatensity for the long exposure time. At higher illuminatiottt the 
10 sensor shows first a non-linear response and will eventually saturate. 

The parameters Xgp, xiow have either preset values, or can vary with time, e.g., be 
determined from earlier output signals and thus adapt to the actual illumination of the 
image sensor 

A preferred example of a merging fonction>7Srux^ for the range for xiow < < Xup is 
IS the following: 

/(x„Xa) = V«?+6-<^K' with c-(xi^xj/(x^-x^) 

This merging fbnction notably exhibits the following desirable properties. It gives 
increased preference to the value xt when xi is close to xi«w* and it gives incrtased 
preference to the value X2 when n is close to Xup. 11 is truly monotonic, continuous 
20 and continuously difFerentiable in Xi and xi, inhibiting banding and discontinuous 
jumps In subsequent algorithms that calculate and employ the first spatial derivative 
of the local brightness values. Moreover^ it is symmetric in xi and x% 
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The above oooBiderations fbr an example with two intarrogatbn runs, le., n » 2, may 
be generalized fbr the case where n ^ 2. Then a mergbg tanjBtionJ(xu'.,,Xn) la used 
fbr combiAifig tt output values Pu ... into a »iigie value Poot according to the 
formula 

where t%, ... , ^ are q>piopriate scaling factors. 

The merging function can be evaluated either in a general-purpose digital 
computation unit, a dedicated digital or analog computaiicm unit that is preferentially 
placed on the same chip as the sensor, or the merging ftinction can be realised with a 
10 iookup-table, either partially or compl^ely. 

The image sensor according to the invention comprises an active area contiuning a 
plurality of pixels, whereby at least two subsets of pbcels may be individually 
interrogated, ix., they can individually be reset and read out. The image sensor 
further comprises means for individually interrogating subsets of pixels, means for 
15 combining output valu^ of said subsets into combined output values, and means for 
dcctricaUy outputting said combined outinit values. 



In the following, the invention and a preferred embodimem thereof are described in 
more detail with t^ference to the drawings, wherein; 

Fig. 1 shows the output of one pixel in the method acccHrding to the invention; 

20 Fig. 2 shows the architecture of an image sensor suited for the method according 
to the invention; 
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Fig.3 shows a flow chart ofthe method acconUng to the iQvcf^ 

Fig. 4 shows an output diamcteristic of a muhiple-exposure image sensor 
according to the invention. 



Figure 1 schemadcaily shows the output P of one given pixel as a fbndion of time T 
3 in the merthod according to the invention. The figure illustrates an example in which 
two e?q}osur€s, a first exposure 31 with a long exposure time 7i«ig and a second 
exposure 34 with a short exposure time rthom are permed on each subset of pixels. 
Afker the first exposure 31, the pixel output vaiue is read out 32 and res^ 33 for a 
first time. A second read-out 3S and, optionally, a second reset 36 are pedbrmed 
10 after the second exposure 34. Then, the i»xel is ready for a fUrther interrogation 
cycle. Of course, other variants of the method are possible in which, e.g., the first 
exposure time is shorter and the second one is longer, or in which more than two 
exposures are perform^ in one interrogadcn cycle. 



Figure 2 shows a schematic of the architecture of an image sensor suited for carrying 
IS out the method according to the invention. In this exemplified embodiment, the 
image sensor has an active area 1 consisting of a matrix of 256x256 pixels 10. 10\ ... 
arranged in rows 11, IT, ... and columns 12> \2\ ... A row addresa decoder 2 and a 
react address decoder 3 are provided for the rows 11, IT, ... The reset address 
decoder has the function of an electronic row shutter. A cdumn driver 4 and a 
20 column address decoder 5 are provided for the columns 12, \2\ ... The image signals 
are outputted via a video output 6, 



Referring to the image sensor of Fig. 2, Figure 3 shows a flow chart of a preferred 
embodiment of the method according to the state of the art. A read-out sequence for 
one full fi'ame with two exposures (cf Rg. 1) is shown. In this embodiment> the 



-.■.■%'.\WJ\WS%VVV.\N 



P1462EP 



November IS, 1998 



-10- 



subseta of pixels are the 2S6 rows 1 1. 1 1', of the image sensor, each subset thus 
consisting of 256 pixel* 10, 10', ... The first subset to be interrogated is, o-g., row No. 
255. A first reset command 21 is issued to ell pixels in row No, 255. After a first, 
long exposure time 22, the output vahies to /gS**" ^56 pixels in this 

S row No, 255 sro read out 23 and immediately stored 24 in analog or digital fimn. 
Immediately afterwards, row No. 255 is reset 25 and exposed during a second, short 
exposure time 26. The output values 1^'^^ to PS^'^^ of aU 256 pixels are read out 

27. This information is either again stored in analog or digital fashion, or it can 
immediately be proo^sed 2S together with the previously stored output vahies 

10 -PtS?^^ to F^^-^\ After row No. 255 has been reset and read twice, the next row 
No. 254 is interrogated in the same fashion; the corresponding dements in Fig. 3 are 
designated with the same reference numbers as for row No. 255 and additional 
i^ostiophes, e.g., reference number 21' stands for a first reset command to all pixels 
in row. No. 254. These steps are performed for each row» ending with row No. 0. 

15 Then each pixel has been read out once, in two interrogation runs, and a first 
complete image can be built up 29 from the processed output vahiea P^^^ to P^J. 
This procedure can be repeated as niany times as desired. 

Figure 4 schematically shows an output characteristic of a nmltiple-exposure image 

sen8or-sccoFding-to-the-inventiOtti"iTe.s-ouQnrt-value8 -E~as-f^^ 

20 intensity /. Again, an exemplified embodimem with two exposures is discussed. A 
first graph 41 in Fig. 4 represents the output Pkmg fer the long exposure. For low 
Intensities, P]mg increases Uncarly with the intensity /; above an upper limit Xj^ it 
flattens due to saturation. A second graph 42 in Fig. 4 represents output P^ for the 
short exposure, multiplied with a scaling factor /, i.e., t^P^ai^ Beneath a lower limit 
25 xiow, the signal is too weak and therefore useless; for higher imensities, t-P^h^ 
increases linearly with the intensity /, A third graph 43 in Fig. 4 shows an example 
for a combined output 
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The merBing fyw6onffxu»i) has the following properties: 

(ii) /<i;i,3Citl"JC5t forxi ^jifup, whereO£xiow<^t9 

5 (iii) >?^iiX^ increased truly monotonlcally 

inxiandjrj fbrxkow<xi <xbp 

Tlus merging function gives increased prefwence to the value xi when Jti is close to 
Xhwt and tt gives increased preftrence to the value vthtn x\ is close to Thus the 
combined output P^a exhibits a smooth transition from the long-exposure range to 
10 the scaled short-exposure range. 



This mvention is not limited to the preferred embodiments described above» to which 
variations and improvements may be made, without departing from the scope of 
protection of the present patent 
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1. Method for inning an image by means of an image sensor with an active area 
(1) containing a phirality of pixets (10, 10\ comprising 

a) in A interrogation runs perfi^rmed on a first subset (1 1) of pixels, where n is 
S an integer and n 2 2, resetting (21, 23) the first subset of pixels (11)» 

exposing (22» 25) the first subset (11) of pbcels and reading out (23, 27) the 
output value(s) (£2^' , ) of the first subset (1 1) of pixels; 

b) combining said output values (P^^\P^^) into a first combined output 
value (£S*'); 

10 c) repeating steps a) and b) for at least one second subset (1 1 ') of pixels. 



2. ^fethod according to claim 1, wher^n said subsets are rows (11, U\ .,.), 
cc^tntis (12, 12\ ,„) or single pixels (10, 10', ...) of the image sensor. 

3. Method according to claim 1 or 2 wherein prior to step a), the active area (1) of 
the image sensor is partitioned into subsets (1 1, 1 V, ...) with equal numbers (256) 

15 ofpixels(10, 10\..). 



4. Method according to one of the claims 1-3, wherein in step a), at least one of said 
output vahie(s) (fjg^ ) ig/«re stored (24, 24\ ...). 

5. Method according to one of the claims wherein steps a) and b) are repeated 
until each pixel (10, 10\ ...) has been read out at leaat once. 
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6, Method according to one of the daints 1 -5, wherein the processing of one subset 
(1 1) of pixels temporally overlaps with the processing of the fbUowmg subset 
(ir) of pixel! 



7. Method according to one of the claims 1*6, wherein /a » 2 for all subsets (II, I T, 
5 of pixels. 



8. Method according to claim 7, whereia in step a), a longer expoaitrc (22, H\ 
and a shorter exposure (26» 16\ ...) are performed. 



9. Method according to claim 8, wherein durtqg said longer exposure (22) of one 
subset (1 IX steps b) are perfonned for all other subsets (IT, ...). 



10 10, Method according to one of the claims 1*9, wherein in step b), said output vahaes 
iPu fPn) are combined into a combined output value (P«Bt) by means of a 
merging flinction (f(xu,.,,Xf^) which is truly monotonic, continuous and 
continuously difFerentiable in ail said output values (Pu,. J^„y 



11. Method according to claim 10 and one of the claims 8 or 9, wherein said meiging 
15 function (ffxi,x%)) has the following properties; 

(i) Preference is given to the output value (X|) obtained firom the loiter 
exposure (22» 22\ when said output values (xt, x^) <^ a combination of 
said output vahies ((xi + X)V2) lie beneath a given lower limit (xiow); 
(vi) Preference is given to the output value (xi) obtained from the shorter 
20 exposure (26, 26\ when said output values (xu x%) or a combination of 

said output values {(x\ xi^tT) lie above a given upper limit (Xup); 
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(iii) Said merging fbnetton (/r^i,x%^) increases truly mcmotonically in said 
output values (jti, when said output values (xu xt) lie between said 
lower limit (jvu) and said uppn* limit ix^p). 

12. Method according to claim 1 1, wheretn said metging (bnction {ffx^x^) i» defined 
5 by 

/(^.3fa)«V^?+(l"^>f? with ^«(«i-xj/(x^-jpj 

13. Method according to one of the claims 1.12, wborein said image sensor is an 
active pixel sensor (APS). 

10 14. Method according to one of the claims 1-13, wherein said output vahies (Pu^ 
Pibxd «^ combined using a general-purpose digital computation unit, a dedicated 
digital or analog computation unit or a lookup table. 

15rlmage sensor for perfoirmin^ 

an active area (1) containing a plurality of pixels (10. 10\ . ,.), whereby at least 
1 5 two subsets (1 1, 1 T, .») of pixels allow an individual interrogation; 

means (2, 3) for individually interrogating subsets (11,11*,-..) of pixels; 
means for combining output values (P^\t^) of said subsets (11. ir. ...) 
into combined output values (£^^ ); and 

means (6) for electrically outputting said combined output values (f^f ^ )- 
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16. Image sensor according to olaim IS, said image sensor being an active pbcel 
sensor (APS). 





Fig. 4 
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ABSTRACT 



The method for forming an image with a wide dynamic range makes use of an image 
sensor containing subsets of pixels that can be individually reset. After an initial reset 
(21), a pixel or row of pixels is exposed (22) for a first time interval and the gray 
S vahie(8) (E^^^ ) are read out (23) and stored (24). The pixel or row of pi?rels is then 
reset (2S) and exposed (26) for a second, shorter time interval. The second gray 
vahie(s) (£^^ ) is/are read out (27) and either stored or immediately combined (28) 
with the first gray value(s) (£2^^) by means of a merging function The merging 

fonction (/) ensures a monotonia smooth ohan^ in output firom th^ lowest to the 
10 highest gray values. The procedure is repeated for all pixels or rows of pixels in the 
image sensor, thus obviating the need for the storage of complete images. The 
method reduces temporal aliasing to a minimum and eliminates spatial aliasing. 



(Figure 3) 
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□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents wUl not correct tlie image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



